Pinpoint Clustering of Web Pages and Mining Implicit Crossover Concepts

In this chapter, we present our Top-N methods for extracting clusters of documents which have originated from the article (Haraguchi, 2002). We first discuss a method for pinpoint clustering of Web pages by pseudo-clique search (Haraguchi & Okubo, 2006; Okubo et al., 2005) and then a method for finding implicit page groups (clusters) represented as formal concepts (Li et al., 2008). A huge collection of documents including pages over theWeb has been considered as an information source of knowledge. One of the core tasks of Information Retrieval (IR) is to effectively find useful and important documents from such a collection. For this purpose, many retrieval engines compute ranks of documents and show them in the order of their ranks (Page et al., 1999; Salton & McGill, 1983). Highly ranked documents are easily checked by users, while documents ranked lower are rarely examined. Any retrieval system based on document ranking has its own ranking scheme. So, even potentially interesting documents are sometimes ranked lower and are therefore actually hidden and invisible to users. In this sense, we might be missing many useful documents. If we can make such hidden significant documents visible, our chance to obtain valuable information and knowledge can be enhanced. The standard approach to cope with this problem is to use the techniques of clustering (Gan et al., 2007) by which we classify various documents into several clusters of similar documents. We pick up a few clusters that seem to be relevant, and then examine them in details to look for interesting documents. However, if the number of clusters is small, clusters tend to be larger ones involving even non-similar documents, and are hard to be examined. Conversely, if we have many clusters, it is also hard to check every cluster, although each cluster is smaller and involves only similar documents. Thus, it is not an easy task to have an adequate method for controlling the number of clusters. This has motivated us to investigate a new clustering method, Pinpoint Clustering, by which we can efficiently extract only nice clusters. We have developed some strategy in (Haraguchi & Okubo, 2006; Okubo et al., 2005) for finding only Top-N number of clusters of similar documents with respect to their evaluation values reflecting the ranks of documents in them. In the framework, the document similarity is evaluated with the help of Singular Value Decomposition(SVD) (Strang, 2003). We first extract semantic correlations among terms by applying SVD to the term-document matrix generated from a corpus with a specific topic. Then, given a set of ranked Web pages to be clustered, we evaluate potential similarities among 19